PublishersREGULARIZATION TOOLS FOR TRAININGFEED - FORWARD NEURAL NETWORKSPART II : Large

We describe regularization tools for training large-scale artiicial feed-forward neural networks. In a companion paper (in this issue) we give the basic ideas and some theoretical results regarding the Gauss-Newton method compared to other methods such as the Levenberg-Marquardt method applied on small and medium size problems. We propose algorithms that explicitly use a sequence of Tikhonov regularized nonlinear least squares problems. For small-and-medium size problems the Gauss-Newton method is applied to the regularized problem. For large-scale problems, methods using new special purpose automatic diierentiation are used in a conjugate gradient method for computing a truncated Gauss-Newton search direction. The algorithms developed utilize the structure of the problem in diierent ways and perform much better than the Polak-Ribi ere based method. All algorithms are tested using benchmark problems and guidelines by Lutz Prechelt in the Proben1 package. All software is written in Matlab and gathered in a toolbox.